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Counting 


Multiplication Rule 


Let's say we have a compound experiment (an experiment with 
multiple components). If the 1st component has nı possible outcomes, 
the 2nd component has n2 possible outcomes, ..., and the rth 
component has n, possible outcomes, then overall there are 
nina..." possibilities for the whole experiment. 


Sampling Table 
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The sampling table gives the number of possible samples of size k out 
of a population of size n, under various assumptions about how the 
sample is collected. 


| Order Matters Not Matter 
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Naive Definition of Probability 


If all outcomes are equally likely, the probability of an event A 
happening is: 


number of outcomes favorable to A 


Praive(A) = 


number of outcomes 


Thinking Conditionally 


Independence 


Independent Events A and B are independent if knowing whether 
A occurred gives no information about whether B occurred. More 
formally, A and B (which have nonzero probability) are independent if 
and only if one of the following equivalent statements holds: 


P(AN B) = P(A)P(B) 
P(A|B) = P(A) 
P(B|A) = P(B) 
Conditional Independence A and B are conditionally independent 
given C if P(AN B|C) = P(A|C)P(B|C). Conditional independence 
does not imply independence, and independence does not imply 
conditional independence. 


Unions, Intersections, and Complements 


De Morgan's Laws A useful identity that can make calculating 
probabilities of unions easier by relating them to intersections, and 
vice versa. Analogous results hold with more than two sets. 


(AU B)* = A^ n BS 
(An B)? = ASU BS 


Joint, Marginal, and Conditional 
Joint Probability P(ANM B) or P(A, B) - Probability of A and B. 
Marginal (Unconditional) Probability P(A) - Probability of A. 


Conditional Probability P(A|B) — P(A, B)/P(B) - Probability of 
A, given that B occurred. 


Conditional Probability is Probability P(A|B) is a probability 
function for any fixed B. Any theorem that holds for probability also 
holds for conditional probability. 


Probability of an Intersection or Union 
Intersections via Conditioning 
P(A, B) = P(A)P(B|A) 
P(A, B, C) = P(A)P(B|A)P(C|A, B) 
Unions via Inclusion- Exclusion 
P(AU B) = P(A) + P(B) — P(An B) 
P(AU BUC) = P(A) + P(B) + P(C) 
— P(AN B)— P(AnC)— P(BnC) 
-P(AnBnC). 


Simpson's Paradox 
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It is possible to have 
P(A| B,C) < P(A| B^,C) and P(A | B,C^) < P(A | B^,C^) 
yet also P(A | B) > P(A | B^). 


Law of Total Probability (LOTP) 


Let B1, Bo, Ba, ...Bn be a partition of the sample space (i.e., they are 
disjoint and their union is the entire sample space). 


P(A) = P(A|Bi)P(B1) + P(A[B2) PCB2) +++ + P(AIBn)P(Bn) 
P(A) = P(AN B1) + P(An B2) +--+ P(AN Bn) 

For LOTP with extra conditioning, just add in another event C! 
P(A|C) = P(A|B1,C)P(Bi|C) +--+ + P(A|Bn, C)P(Bn|C) 
P(A|C) = P(AN B1|C) + P(AN B3|C) t---- -- P(AN B,|C) 

Special case of LOTP with B and B^ as partition: 

P(A) = P(A|B)P(B) + P(A[B^)P(B^) 
P(A) = P(An B) - P(An B^) 


Bayes’ Rule 
Bayes’ Rule, and with extra conditioning (just add in C!) 


P(A|B) = FOU 
nao. ge 


We can also write 
P(A, B,C) u P(B,C|A)P(A) 


EUM P(B,C) P(B,C) 
Odds Form of Bayes’ Rule 
P(A|B) _ P(BIA) P(A) 
P(A*|B) | P(B|A*) P(A*) 


'The posterior odds of A are the likelihood ratio times the prior odds. 


Random Variables and their Distributions 


PMF, CDF, and Independence 


Probability Mass Function (PMF) Gives the probability that a 
discrete random variable takes on the value x. 


px (a) = P(X = x) 
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The PMF satisfies 


px (x) > 0 and So px (2) =1 


Cumulative Distribution Function (CDF) Gives the probability 
that a random variable is less than or equal to x. 


Fx (x) = P(X <2) 
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The CDF is an increasing, right-continuous function with 


Fx(x)— 0 as x — —oo and Fx (x) + 1 as x — oo 


Independence Intuitively, two random variables are independent if 
knowing the value of one gives no information about the other. 
Discrete r.v.s X and Y are independent if for all values of x and y 


P(X =2,Y =y) = P(X =2)P(Y = y) 


Expected Value and Indicators 


Expected Value and Linearity 


Expected Value (a.k.a. mean, expectation, or average) is a weighted 
average of the possible outcomes of our random variable. 
Mathematically, if 71, £2, %3,... are all of the distinct possible values 
that X can take, the expected value of X is 


X Y X+Y 
3 4 y] 
2 2 4 
6 8 14 
10 23 33 
1 -3 -2 
1 0 1 
5 9 14 
4 1 5 


xà + pÈ» = sMOw» 
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E(X) $ E(Y) = 
Linearity For any r.v.s X and Y, and constants a, b, c, 


E(aX + bY +c) 2 aE(X) - bE(Y) +c 


E(X +Y) 


Same distribution implies same mean If X and Y have the same 
distribution, then E(X) — E(Y) and, more generally, 


E(g(X)) = E(g(Y)) 


Conditional Expected Value is defined like expectation, only 
conditioned on any event A. 


E(X|A) = X 2P(X = 2A) 
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Indicator Random Variables 


Indicator Random Variable is a random variable that takes on the 
value 1 or 0. It is always an indicator of some event: if the event 
occurs, the indicator is 1; otherwise it is 0. They are useful for many 
problems about counting how many events of some kind occur. Write 


1 
IA = 


Note that ry =T1,4,lalp = IAnp, and Iausg = IA + Ip — Ials. 
Distribution I4 ~ Bern(p) where p = P(A). 


if A occurs, 
if A does not occur. 


Fundamental Bridge The expectation of the indicator for event A is 
the probability of event A: E(I4) = P(A). 


Variance and Standard Deviation 
Var(X) = E (X — E(X)? = E(X’) - (B(X))? 


SD(X) = /Var(X) 


Continuous RVs, LOTUS, UoU 


Continuous Random Variables (CRVs) 


What’s the probability that a CRV is in an interval? Take the 
difference in CDF values (or use the PDF as described later). 


P(a < X <b) = P(X < b) - P(X € a) = Fx (b) — Fx (a) 


For X ~ N (p, 07), this becomes 
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What is the Probability Density Function (PDF)? The PDF f 
is the derivative of the CDF F. 


F'() = f(z) 


A PDF is nonnegative and integrates to 1. By the fundamental 
theorem of calculus, to get from PDF back to CDF we can integrate: 


F(a) =f" roa 
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To find the probability that a CRV takes on a value in an interval, 
integrate the PDF over that interval. 


b 
F(b) — F(a) -f f(x)dx 


How do I find the expected value of a CRV? Analogous to the 
discrete case, where you sum x times the PMF, for CRVs you integrate 
x times the PDF. 


LOTUS 


Expected value of a function of an r.v. The expected value of X 
is defined this way: 


E(X) = 5 xP(X = x) (for discrete X) 


x 


E(X) = D xf(x)dx (for continuous X) 


The Law of the Unconscious Statistician (LOTUS) states that 
you can find the expected value of a function of a random variable, 
g( X), in a similar way, by replacing the x in front of the PMF/PDF by 
g(x) but still working with the PMF/PDF of X: 


E(g(X)) — b» g(x)P(X — x) (for discrete X) 
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E(g(X)) = us g(x)f(x)d« (for continuous X) 


What's a function of a random variable? A function of a random 
variable is also a random variable. For example, if X is the number of 
bikes you see in an hour, then g(.X) — 2X is the number of bike wheels 


you see in that hour and h(X) = (3) a a is the number of 
pairs of bikes such that you see both of those bikes in that hour. 


What’s the point? You don’t need to know the PMF/PDF of g(X) 
to find its expected value. All you need is the PMF/PDF of X. 


Universality of Uniform (UoU) 


When you plug any CRV into its own CDF, you get a Uniform(0,1) 
random variable. When you plug a Uniform(0,1) r.v. into an inverse 
CDF, you get an r.v. with that CDF. For example, let’s say that a 
random variable X has CDF 


F(x)=1—e 7, forz>0 


By UoU, if we plug X into this function then we get a uniformly 
distributed random variable. 


F(X) =1-—e * ~ Unif(0, 1) 
Similarly, if U ~ Unif(0, 1) then F7'(U) has CDF F. The key point is 


that for any continuous random variable X, we can transform it into a 
Uniform random variable and back by using its CDF. 


Moments and MGFs 


Moments 


Moments describe the shape of a distribution. Let X have mean p and 
standard deviation c, and Z = (X — w)/o be the standardized version 
of X. The kth moment of X is up = E(X") and the kth standardized 


moment of X is mk = E(Z*). The mean, variance, skewness, and 
kurtosis are important summaries of the shape of a distribution. 


Mean E(X) = pı 
Variance Var(X) = p2 — u? 
Skewness Skew(X) = m3 


Kurtosis Kurt(X) = m4 — 3 


Moment Generating Functions 
MGF For any random variable X, the function 
Mx(t) = E(e**) 


is the moment generating function (MGF) of X, if it exists for all 
t in some open interval containing 0. The variable t could just as well 
have been called u or v. It's a bookkeeping device that lets us work 
with the function Mx rather than the sequence of moments. 


Why is it called the Moment Generating Function? Because 
the kth derivative of the moment generating function, evaluated at 0, 
is the kth moment of X. 


ui = E(X") = MY (0) 
This is true by Taylor expansion of e** since 
Sc E(X*)^ — Y ukt? 
k=0 k! kco k! 
MGF of linear functions If we have Y = aX + b, then 
My (t) = E(e! (** *9) = e” E(e®®)X) = e*t Mx (at) 


Mx (t) 2 E(e**) = 


Uniqueness Jf it exists, the MGF uniquely determines the 
distribution. This means that for any two random variables X and Y, 
they are distributed the same (their PMFs/PDFs are equal) if and 
only if their MGFs are equal. 


Summing Independent RVs by Multiplying MGFs. If X and Y 
are independent, then 


Mx+y (t) = E(ef V * Y) ) = g(e*X)g(e*Y ) = Mx (t)- My (t) 


The MGF of the sum of two random variables is the product of the 
MGFs of those two random variables. 


Joint PDFs and CDFs 


Joint Distributions 
The joint CDF of X and Y is 
F(x,y) = P(X < x,Y € y) 

In the discrete case, X and Y have a joint PMF 

px,y (x,y) = P(X =2,Y =y). 
In the continuous case, they have a joint PDF 

a2 

fx,y(@,y) = Izay Y OY): 

The joint PMF/PDF must be nonnegative and sum/integrate to 1. 


Conditional Distributions 
Conditioning and Bayes' rule for discrete r.v.s 

P(X =2,Y =y) _ P(X =al¥ -y)P(Y =y) 
“P(x =a) P(X = 2) 
Conditioning and Bayes’ rule for continuous r.v.s 
fx,y (x,y) fxiv (aly) fv (v) 

fx (x) fx (x) 


P(Y =y|X = x) 


fy|x(ylz) = 


Hybrid Bayes’ rule 
P(A|X = 2) fx (v) 
P(A) 


fx(v|A) = 


Marginal Distributions 


To find the distribution of one (or more) random variables from a joint 


PMF/PDF, sum/integrate over the unwanted random variables. 


Marginal PMF from joint PMF 


P(X =2) =) P(X =2,Y =y) 
y 
Marginal PDF from joint PDF 
fx (2) =f fx,y (x, y)dy 


Independence of Random Variables 


Random variables X and Y are independent if and only if any of the 
following conditions holds: 


e Joint CDF is the product of the marginal CDFs 

e Joint PMF/PDF is the product of the marginal PMFs/PDFs 

e Conditional distribution of Y given X is the marginal 
distribution of Y 


Write X JL Y to denote that X and Y are independent. 
Multivariate LOTUS 


LOTUS in more than one dimension is analogous to the 1D LOTUS. 
For discrete random variables: 


E(g(X,Y)) =X S gm, y)P(X = 2, Y = y) 
Toy 
For continuous random variables: 


E(g(X,Y)) = L = f = eater eaid 


Covariance and Transformations 


Covariance and Correlation 
Covariance is the analog of variance for two random variables. 
Cov(X, Y) = E((X — E(X))(Y — E(Y))) = E(XY) — E(X)E(Y) 


Note that 
Cov(X, X) = E(X?) — (E(X))? = Var(X) 


Correlation is a standardized version of covariance that is always 


between —1 and 1. 

Cov( X,Y 

Corr(X, Y) = ON) 
Var(X)Var(Y) 


Covariance and Independence If two random variables are 
independent, then they are uncorrelated. The converse is not 
necessarily true (e.g., consider X ~ A/(0, 1) and Y = X?). 


X IL Y — Cov(X,Y) = 0 — E(XY) = E(X)E(Y) 
Covariance and Variance The variance of a sum can be found by 
Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y) 
Var(X1 + Xa +- Xn) = Y^ va) +2) Cov(X;, X;) 
i=1 i<j 
If X and Y are independent then they have covariance 0, so 
XIL Y Var(X + Y) = Var(X) + Var(Y) 


If X1, X2,..., Xn are identically distributed and have the same 
covariance relationships (often by symmetry), then 


Var(Xı + Xa +- + Xn) = nVar(X1) + (7) Cov(X1, X2) 


Covariance Properties For random variables W, X, Y, Z and 
constants a, b: 


Cov(X, Y) = Cov(Y, X) 
Cov(X + a, Y + b) = Cov(X, Y) 
Cov(aX,bY) = abCov(X, Y) 
Cov(W + X,Y + Z) = Cov(W, Y) + Cov(W, Z) + Cov(X, Y) 
+ Cov(X, Z) 


Correlation is location-invariant and scale-invariant For any 
constants a, b, c, d with a and c nonzero, 


Corr(a.X + b, cY + d) = Corr(X, Y) 


Transformations 


One Variable Transformations Let’s say that we have a random 
variable X with PDF fx (a), but we are also interested in some 
function of X. We call this function Y = g(X). Also let y = g(x). If g 
is differentiable and strictly increasing (or strictly decreasing), then 
the PDF of Y is 


fv(y) = fx (v) 


dx Sä d. 4 
=| =fx(g (wy) a w| 
y dy 
The derivative of the inverse transformation is called the Jacobian. 
Two Variable Transformations Similarly, let’s say we know the 


joint PDF of U and V but are also interested in the random vector 
(X, Y) defined by (X,Y) = g(U,V). Let 


be the Jacobian matrix. If the entries in this matrix exist and are 
continuous, and the determinant of the matrix is never 0, then 


'The inner bars tells us to take the matrix's determinant, and the outer 
bars tell us to take the absolute value. In a 2 x 2 matrix, 


fx,v (vy) = fuv (u,v) 


oen 
O(a, y) 


| = |ad — be| 


Convolutions 


Convolution Integral If you want to find the PDF of the sum of two 
independent CRVs X and Y, you can do the following integral: 


fx+y(t) = I. fx (x) fy (t — z)dx 


Example Let X,Y ~ N (0,1) be i.i.d. Then for each fixed t, 
es 1 
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By completing the square and using the fact that a Normal PDF 
integrates to 1, this works out to fx... y (t) being the AN (0,2) PDF. 


Poisson Process 


Definition We have a Poisson process of rate A arrivals per unit 
time if the following conditions hold: 
1. The number of arrivals in a time interval of length t is Pois(At). 
2. Numbers of arrivals in disjoint time intervals are independent. 
For example, the numbers of arrivals in the time intervals [0, 5], 
(5, 12), and [13, 23) are independent with Pois(5A), Pois(7A), Pois(10A) 
distributions, respectively. 
H * X—*X X—X—» 
0 T, T, T. Tw. 


1 2 3 4 5 


Count-Time Duality Consider a Poisson process of emails arriving 
in an inbox at rate A emails per hour. Let Tn be the time of arrival of 
the nth email (relative to some starting time 0) and N; be the number 
of emails that arrive in [0, t]. Let's find the distribution of Tı. The 
event T, > t, the event that you have to wait more than t hours to get 
the first email, is the same as the event Nz = 0, which is the event that 
there are no emails in the first t hours. So 


P(T, > t) = P(N; 20) 2e P(T, < t)=1-—e7™ 


Thus we have Tı ~ Expo(A). By the memoryless property and similar 
reasoning, the interarrival times between emails are i.i.d. Expo(A), i.e., 
the differences T, — T,—1 are i.i.d. Expo(A). 


Order Statistics 


Definition Let's say you have n i.i.d. r.v.s X4, X5,..., Xn. If you 
arrange them from smallest to largest, the ith element in that list is 
the ith order statistic, denoted X(;). So X(1) is the smallest in the list 
and X(,) is the largest in the list. 


Note that the order statistics are dependent, e.g., learning X(4) = 42 
gives us the information that X(1), X(2), X(3) are < 42 and 
X(5). X(o); EE. Xin) are > 42. 


Distribution Taking n i.i.d. random variables X1, X2,.. 
CDF F(a) and PDF f(x), the CDF and PDF of X(;) are: 


.,Xn with 


Fx (2) = P(Xo) € 2) = » (JFera-Fey" 
fx) = nC 7 :) F(z)/-!(1— F(z))"- f(a) 


Uniform Order Statistics The jth order statistic of 
iid. Uj,... , Un ~ Unif(0, 1) is Uj) ~ Beta(j, n — j + 1). 


Conditional Expectation 


Conditioning on an Event We can find E(Y|A), the expected value 
of Y given that event A occurred. A very important case is when A is 
the event X = x. Note that E(Y |A) is a number. For example: 


e The expected value of a fair die roll, given that it is prime, is 
-2+4 3+5 5#. 


e Let Y be the number of successes in 10 independent Bernoulli 
trials with probability p of success. Let A be the event that the 
first 3 trials are all successes. Then 


E(Y|A) 2 3-- 7p 


since the number of successes among the last 7 trials is Bin(7, p). 


e Let T ~ Expo(1/10) be how long you have to wait until the 
shuttle comes. Given that you have already waited t minutes, 
the expected additional waiting time is 10 more minutes, by the 
memoryless property. That is, E(T|T > t) = t+ 10. 


Discrete Y Continuous Y 


EY) - X, yP(Y =) 


E(Y|A) = Z, yP(Y = ylA) 


E(Y) = f, ufv (y)dy 


E(Y|A) = [Zo uf(y| A)dy 


Conditioning on a Random Variable We can also find E(Y|X), 
the expected value of Y given the random variable X. This is a 
function of the random variable X. It is not a number except in 
certain special cases such as if X lL Y. To find E(Y|X), find 
E(Y|X = x) and then plug in X for x. For example: 


e If E(Y|X = z) = a? + 5x, then E(Y|X) = X? - 5X. 


e Let Y be the number of successes in 10 independent Bernoulli 
trials with probability p of success and X be the number of 
successes among the first 3 trials. Then E(Y|X) = X + 7p. 


e Let X ~ N (0,1) and Y = X?. Then E(Y|X = z) = z? since if 
we know X = z then we know Y = x”. And E(X|Y = y) 2 0 
since if we know Y = y then we know X = +,/y, with equal 


probabilities (by symmetry). So E(Y|X) = X?, E(X|Y) — 0. 


Properties of Conditional Expectation 


1. E(Y|X) 2 E(Y) if X LY 


2. E(h(X)W|X) = h(X)E(W|X) (taking out what's known) 
In particular, E(h(X)|X) = h(X). 


3. E(E(Y|X)) = E(Y) (Adam's Law, a.k.a. Law of Total 
Expectation) 


Adam's Law (a.k.a. Law of Total Expectation) can also be 
written in a way that looks analogous to LOTP. For any events 
A1, À2,..., An that partition the sample space, 
E(Y) = E(Y|A1) P(A1) t: + B(Y|An)P(An) 
For the special case where the partition is A, A^, this says 
E(Y) = E(Y|A)P(A) + E(Y|A*)P(A^) 
Eve's Law (a.k.a. Law of Total Variance) 


Var(Y) = E(Var(Y|X)) + Var(E(Y|X)) 


MVN, LLN, CLT 


Law of Large Numbers (LLN) 
Let X1, X2, Xa... be i.i.d. with mean u. The sample mean is 


x Xı +X2+X3 + +Xn 


E 


n 


The Law of Large Numbers states that as n — oo, Xn — u with 
probability 1. For example, in flips of a coin with probability p of 
Heads, let Xj be the indicator of the jth flip being Heads. Then LLN 
says the proportion of Heads converges to p (with probability 1). 


Central Limit Theorem (CLT) 
Approximation using CLT 


We use ~ to denote is approximately distributed. We can use the 
Central Limit Theorem to approximate the distribution of a random 
variable Y = X, + X2 +--+ Xn that is a sum of n i.i.d. random 
variables X;. Let E(Y) = py and Var(Y) = oł}. The CLT says 


YON (uy, oy) 


If the X; are i.i.d. with mean ux and variance ox, then wy = nux 
and oy z nox. For the sample mean Xn, the CLT says 


= 1 
Xn = (Kit Xa ted Xn) & N (ux, o% /n) 


Asymptotic Distributions using CLT 


We use + to denote converges in distribution to as n — oo. The 
CLT says that if we standardize the sum X +--- + Xn then the 
distribution of the sum converges to N (0, 1) as n — oo: 


1 
oyn 


In other words, the CDF of the left-hand side goes to the standard 
Normal CDF, ®. In terms of the sample mean, the CLT says 


vn(Xs p Ax) D 


(Xi e Xn nix) 2+ N(0,1) 


— N (0,1) 
ox 
Markov Chains 
Definition 
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A Markov chain is a random walk in a state space, which we will 
assume is finite, say (1,2,..., M}. We let X, denote which element of 
the state space the walk is visiting at time t. The Markov chain is the 
sequence of random variables tracking where the walk is at all points 
in time, Xo, X1, X2,.... By definition, a Markov chain must satisfy 
the Markov property, which says that if you want to predict where 
the chain will be at a future time, if we know the present state then 
the entire past history is irrelevant. Given the present, the past and 
future are conditionally independent. In symbols, 


P(Xn41 = j| Xo = io, X1 = à,..., Xn = i) = P(Xn41 = 3| Xa — à) 


State Properties 
A state is either recurrent or transient. 


e If you start at a recurrent state, then you will always return 
back to that state at some point in the future. 9 You can 
check-out any time you like, but you can never leave. à 


e Otherwise you are at a transient state. There is some positive 
probability that once you leave you will never return. à You 
don’t have to go home, but you can't stay here. à 


A state is either periodic or aperiodic. 


e If you start at a periodic state of period k, then the GCD of 
the possible numbers of steps it would take to return back is 
k>1. 


e Otherwise you are at an aperiodic state. The GCD of the 
possible numbers of steps it would take to return back is 1. 


Transition Matrix 


Let the state space be (1,2,..., M). The transition matrix Q is the 
M x M matrix where element q;j is the probability that the chain goes 
from state i to state j in one step: 


qij = P(Xnq1 = j| Xn = i) 


To find the probability that the chain goes from state i to state j in 
exactly m steps, take the (i, j) element of Q™. 


qp? = P(Xnim = j|Xn =i) 


If Xo is distributed according to the row vector PMF p, i.e., 
pj; = P(Xo = j), then the PMF of Xn is PQ”. 


Chain Properties 


A chain is irreducible if you can get from anywhere to anywhere. If a 
chain (on a finite state space) is irreducible, then all of its states are 
recurrent. A chain is periodic if any of its states are periodic, and is 
aperiodic if none of its states are periodic. In an irreducible chain, all 
states have the same period. 


A chain is reversible with respect to s' if siqij = Sjqji for all i, j. 
Examples of reversible chains include any chain with qi; = qji, with 


z—(l 1 1 : 
T= (ar ILL m and random walk on an undirected network. 


Stationary Distribution 


Let us say that the vector 8 — (s1,52,..., s) be a PMF (written as a 
row vector). We will call ï the stationary distribution for the chain 
if SQ = s. As a consequence, if X, has the stationary distribution, 
then all future X41, X:42,... also have the stationary distribution. 


For irreducible, aperiodic chains, the stationary distribution exists, is 
unique, and s; is the long-run probability of a chain being at state i. 
The expected number of steps to return to i starting from i is 1/s;. 


To find the stationary distribution, you can solve the matrix equation 
(Q' — I)8' — 0. The stationary distribution is uniform if the columns 
of Q sum to 1. 


Reversibility Condition Implies Stationarity If you have a PMF s 
and a Markov chain with transition matrix Q, then s;qij = Sjqji for 
all states i,j implies that 8 is stationary. 


Random Walk on an Undirected Network 


If you have a collection of nodes, pairs of which can be connected by 
undirected edges, and a Markov chain is run by going from the 
current node to a uniformly random node that is connected to it by an 
edge, then this is a random walk on an undirected network. The 
stationary distribution of this chain is proportional to the degree 
sequence (this is the sequence of degrees, where the degree of a node 
is how many edges are attached to it). For example, the stationary 
distribution of random walk on the network shown above is 


proportional to (3, 3, 2, 4, 2), so it’s Gk. ip ip ip AD 


Continuous Distributions 


Uniform Distribution 
Let us say that U is distributed Unif(a, b). We know the following: 


Properties of the Uniform For a Uniform distribution, the 
probability of a draw from any interval within the support is 
proportional to the length of the interval. See Universality of Uniform 
and Order Statistics for other properties. 


Example William throws darts really badly, so his darts are uniform 
over the whole room because they're equally likely to appear anywhere. 
William's darts have a Uniform distribution on the surface of the 
room. The Uniform is the only distribution where the probability of 
hitting in any specific region is proportional to the length/area/volume 
of that region, and where the density of occurrence in any one specific 
spot is constant throughout the whole support. 


Normal Distribution 
Let us say that X is distributed N (u, e). We know the following: 


Central Limit Theorem The Normal distribution is ubiquitous 
because of the Central Limit Theorem, which states that the sample 
mean of i.i.d. r.v.s will approach a Normal distribution as the sample 
size grows, regardless of the initial distribution. 


Location-Scale Transformation Every time we shift a Normal 
r.v. (by adding a constant) or rescale a Normal (by multiplying by a 
constant), we change it to another Normal r.v. For any Normal 

X ~ N(u, 07), we can transform it to the standard A/(0, 1) by the 
following transformation: 


X— 
z= H 


~ .N (0, 1) 


Standard Normal The Standard Normal, Z ~ N (0,1), has mean 0 
and variance 1. Its CDF is denoted by ®. 


Exponential Distribution 
Let us say that X is distributed Expo(A). We know the following: 


Story You're sitting on an open meadow right before the break of 
dawn, wishing that airplanes in the night sky were shooting stars, 
because you could really use a wish right now. You know that shooting 
stars come on average every 15 minutes, but a shooting star is not 
*due" to come just because you've waited so long. Your waiting time 
is memoryless; the additional time until the next shooting star comes 
does not depend on how long you've waited already. 


Example The waiting time until the next shooting star is distributed 
Expo(4) hours. Here à = 4 is the rate parameter, since shooting 
stars arrive at a rate of 1 per 1/4 hour on average. The expected time 
until the next shooting star is 1/A = 1/4 hour. 


Expos as a rescaled Expo(1) 
Y ~ Expo(A) > X = AY ~ Expo(1) 


Memorylessness The Exponential Distribution is the only 
continuous memoryless distribution. The memoryless property says 
that for X ~ Expo(A) and any positive numbers s and t, 


P(X >s4+t|X >s) = P(X >t) 
Equivalently, 
X — a|(X > a) ~ Expo(A) 
For example, a product with an Expo(A) lifetime is always “as good as 


new" (it doesn't experience wear and tear). Given that the product has 
survived a years, the additional time that it will last is still Expo(A). 


Min of Expos If we have independent X; ~ Expo(A;), then 
min( Xi, iu Xk) ~ Expo(A1 +2 +--+ Ax). 


Max of Expos If we have i.i.d. X; ~ Expo(A), then 
max(X ,..., X4) has the same distribution as Yı + Y2 +--- + Yk, 
where Y; ^ Expo(jA) and the Y; are independent. 


Gamma Distribution 
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Let us say that X is distributed Gamma(a, A). We know the following: 


Story You sit waiting for shooting stars, where the waiting time for a 
star is distributed Expo(A). You want to see n shooting stars before 
you go home. The total waiting time for the nth shooting star is 
Gamma(n, A). 


Example You are at a bank, and there are 3 people ahead of you. 
The serving time for each person is Exponential with mean 2 minutes. 
Only one person at a time can be served. The distribution of your 
waiting time until it’s your turn to be served is Gamma(3, i A 


Beta Distribution 
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Conjugate Prior of the Binomial In the Bayesian approach to 
statistics, parameters are viewed as random variables, to reflect our 
uncertainty. The prior for a parameter is its distribution before 
observing data. The posterior is the distribution for the parameter 
after observing data. Beta is the conjugate prior of the Binomial 
because if you have a Beta-distributed prior on p in a Binomial, then 
the posterior distribution on p given the Binomial data is also 
Beta-distributed. Consider the following two-level model: 


X|p ~ Bin(n, p) 
p ~ Beta(a, b) 
Then after observing X = x, we get the posterior distribution 
p\(X = x) ~ Beta(a + z, b +n — x) 
Order statistics of the Uniform See Order Statistics. 


Beta-Gamma relationship If X ~ Gamma(a, A), 
Y ~ Gamma(b, A), with X JL Y then 


e. xiy ~ Beta(a, b) 


e X+Y dL X. 


This is known as the bank—post office result. 
x? (Chi-Square) Distribution 


Let us say that X is distributed xi. We know the following: 


Story A Chi-Square(n) is the sum of the squares of n independent 
standard Normal r.v.s. 


Properties and Representations 
X is distributed as Ze + Z2 Tec Ze for i.i.d. Z; ~ N(O, 1) 
X ~ Gamma(n/2, 1/2) 


Discrete Distributions 


Distributions for four sampling schemes 


Replace No Replace 
Fixed # trials (n) Binomial HGeom 
(Bern if n = 1) 
Draw until r success NBin NHGeom 


(Geom if r = 1) 


Bernoulli Distribution 


The Bernoulli distribution is the simplest case of the Binomial 
distribution, where we only have one trial (n = 1). Let us say that X is 
distributed Bern(p). We know the following: 


Story A trial is performed with probability p of “success”, 
the indicator of success: 1 means success, 0 means failure. 


and X is 


Example Let X be the indicator of Heads for a fair coin toss. Then 
X ~ Bern(1). Also, 1 — X ~ Bern($) is the indicator of Tails. 


Binomial Distribution 


Bin(10,1/2) 
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Let us say that X is distributed Bin(n, p). We know the following: 


Story X is the number of "successes" that we will achieve in n 
independent trials, where each trial is either a success or a failure, each 
with the same probability p of success. We can also write X as a sum 
of multiple independent Bern(p) random variables. Let X ~ Bin(n, p) 
and X; ~ Bern(p), where all of the Bernoullis are independent. Then 


X = Xi 4X2 4X8 XA 


Example If Jeremy Lin makes 10 free throws and each one 
independently has a i chance of getting in, then the number of free 
throws he makes is distributed Bin(10, 3 : 


Properties Let X ~ Bin(n,p),Y ~ Bin(m, p) with X IL Y. 


e Redefine success n — X ~ Bin(n,1-— p) 
e Sum X + Y ~ Bin(n + m, p) 


e Conditional X|(X + Y =r) ~ HGeom(n, m,r) 

e Binomial-Poisson Relationship Bin(n, p) is approximately 
Pois(A) if p is small. 

e Binomial-Normal Relationship Bin(n, p) is approximately 
N (np, np(1 — p)) if n is large and p is not near 0 or 1. 


Geometric Distribution 
Let us say that X is distributed Geom(p). We know the following: 


Story X is the number of "failures" that we will achieve before we 
achieve our first success. Our successes have probability p. 


Example If each pokeball we throw has probability is to catch Mew, 
the number of failed pokeballs will be distributed Geom( 5). 


First Success Distribution 


Equivalent to the Geometric distribution, except that it includes the 
first success in the count. This is 1 more than the number of failures. 
If X ~ FS(p) then E(X) = 1/p. 


Negative Binomial Distribution 
Let us say that X is distributed NBin(r, p). We know the following: 


Story X is the number of "failures" that we will have before we 
achieve our rth success. Our successes have probability p. 


Example Thundershock has 6096 accuracy and can faint a wild 
Raticate in 3 hits. The number of misses before Pikachu faints 
Raticate with Thundershock is distributed NBin(3, 0.6). 


Hypergeometric Distribution 


Let us say that X is distributed HGeom(w, b, n). We know the 
following: 


Story In a population of w desired objects and b undesired objects, 
X is the number of “successes” we will have in a draw of n objects, 
without replacement. The draw of n objects is assumed to be a 
simple random sample (all sets of n objects are equally likely). 


Examples Here are some HGeom examples. 


e Let's say that we have only b Weedles (failure) and w Pikachus 
(success) in Viridian Forest. We encounter n Pokemon in the 
forest, and X is the number of Pikachus in our encounters. 


e The number of Aces in a 5 card hand. 


e You have w white balls and b black balls, and you draw n balls. 
You will draw X white balls. 


e You have w white balls and b black balls, and you draw m balls 
without replacement. The number of white balls in your sample 
is HGeom(w, b, n); the number of black balls is HGeom(b, w, n). 


e Capture-recapture A forest has N elk, you capture n of them, 
tag them, and release them. Then you recapture a new sample 
of size m. How many tagged elk are now in the new sample? 
HGeom(n, N — n, m) 


Poisson Distribution 
Let us say that X is distributed Pois(A). We know the following: 


Story There are rare events (low probability events) that occur many 
different ways (high possibilities of occurences) at an average rate of A 
occurrences per unit space or time. The number of events that occur 
in that unit of space or time is X. 


Example A certain busy intersection has an average of 2 accidents 
per month. Since an accident is a low probability event that can 
happen many different ways, it is reasonable to model the number of 
accidents in a month at that intersection as Pois(2). Then the number 
of accidents that happen in two months at that intersection is 
distributed Pois(4). 


Properties Let X ~ Pois(A1) and Y ~ Pois(A2), with X IL Y. 
1. Sum X + Y ~ Pois(A1 + 43) 
2. Conditional X|(X +Y = n) Bin (n. xd) 


3. Chicken-egg If there are Z ~ Pois(A) items and we randomly 
and independently “accept” each item with probability p, then 
the number of accepted items Z; ~ Pois(Ap), and the number of 
rejected items Z2 ~ Pois(A(1 — p)), and Z JL Zo. 


Multivariate Distributions 


Multinomial Distribution 


Let us say that the vector X= (X1, X2, Xa,. 
where p = (p1, p2,---,Dk): 

Story We have n items, which can fall into any one of the k buckets 
independently with the probabilities P = (p1, P2,- -, Pk) 


oo Xy) ~ Mult; (n, p) 


Example Let us assume that every year, 100 students in the Harry 
Potter Universe are randomly and independently sorted into one of 
four houses with equal probability. The number of people in each of the 
houses is distributed Mult4(100, p), where p = (0.25, 0.25, 0.25, 0.25). 
Note that X4 + X2+---+ X4 = 100, and they are dependent. 


Joint PMF For n = nı +7n2+---+7k, 


2 n! 


P(X =f) = 


ni no nk 
n1!n3! ae ae P2 U “Pr 
Marginal PMF, Lumping, and Conditionals Marginally, 

Xj ~ Bin(n, pi) since we can define “success” to mean category i. If 
you lump together multiple categories in a Multinomial, then it is still 
Multinomial. For example, X; + X; ~ Bin(n, pi + pj) for i Z j since 
we can define “success” to mean being in category i or j. Similarly, if 
k — 6 and we lump categories 1-2 and lump categories 3-5, then 


(Xi + X2, Xa + X4 Xs, Xe) ~ Multa(n, (pı + p2, pa + pa + ps, pe)) 


Conditioning on some X; also still gives a Multinomial: 


C m ( pi 2-3) 
l— pk 1— pr 


Variances and Covariances We have X; ~ Bin(n, p;) marginally, so 
Var(X;) = npi(1— pi). Also, Cov(X;, Xj) = —npipj for i Z j. 


Multivariate Uniform Distribution 


See the univariate Uniform for stories and examples. For the 2D 
Uniform on some region, probability is proportional to area. Every 
point in the support has equal density, of value Sres cfregion/ For the 
3D Uniform, probability is proportional to volume. 


Multivariate Normal (MVN) Distribution 


A vector X — (X3, X2,..., Xk) is Multivariate Normal if every linear 
combination is Normally distributed, i.e., £1.X1 + t3.X3 +- 4- ti Xy is 
Normal for any constants t),t2,...,t,%. The parameters of the 
Multivariate Normal are the mean vector ji = (u1,12,..., Hk) and 
the covariance matrix where the (i, j) entry is Cov(X;, X;). 


Properties The Multivariate Normal has the following properties. 


e Any subvector is also MVN. 


e If any two elements within an MVN are uncorrelated, then they 
are independent. 


e The joint PDF of a Bivariate Normal (X, Y) with (0, 1) 
marginal distributions and correlation p € (—1, 1) is 


1 
fx,v (2, y) = > — exp ( (a? +y? 2p2v)), 


2 


with T = 1 — p°. 


Distribution Properties 


Important CDFs 

Standard Normal ©® 

Exponential(A) F(x)—1-— e-?^*, for x € (0,00) 
Uniform(0,1) F(x) = x, for x € (0,1) 


Convolutions of Random Variables 
A convolution of n random variables is simply their sum. For the 
following results, let X and Y be independent. 
1. X ~ Pois(A1), Y ~ Pois(A2) — X + Y ~ Pois(A1 + A2) 
2. X ~ Bin(ni,p), Y ~ Bin(ne,p) —9 X + Y ~ Bin(n; 4 no, p). 
Bin(n, p) can be thought of as a sum of i.i.d. Bern(p) r.v.s. 
3. X ~ Gamma(ai, A), Y ~ Gamma(az, A) 
— X +Y ~ Gamma(a; + a2, A). Gamma(n, A) with n an 
integer can be thought of as a sum of i.i.d. Expo(A) r.v.s. 
4. X ~ NBin(ri, p), Y ~ NBin(rs, p) 
——> X +Y ~ NBin(ri + ro, p). NBin(r, p) can be thought of as 
a sum of i.i.d. Geom(p) r.v.s. 
5. X ~ N (i02), Y ~ N (ua, 0) 
— X +Y e N( + ua,01 4 02) 


Special Cases of Distributions 
1. Bin(l, p) ~ Bern(p) 


2. Beta(1, 1) ~ Unif(0, 1) 

3. Gamma(1, A) ~ Expo(A) 

4. xà ^. Gamma (4, i) 

5. NBin(1, p) ~ Geom(p) 
Inequalities 


1. Cauchy-Schwarz |E(XY)| < / E(X2) E(Y?) 
2. Markov P(X >a) < 2*! fora» 0 
Chebyshev P(|X — u| >a) € 2 for E(X) = n, Var(X) = 0? 


4. Jensen E(g(X)) > g(E(X)) for g convex; reverse if g is 
concave 


Formulas 


Geometric Series 


n =p” 
l+rtrete--4 [pm NC E 
l-r 
k=0 
2 1 
l+trtr4---= if |r| « 1 


Exponential Function (e?) 


Gamma and Beta Integrals 


You can sometimes solve complicated-looking integrals by 
pattern-matching to a gamma or beta integral: 


g9 E Pe aus = T (a)r (b 
f z' le * dx =T (t) fÍ z^ l(1—x)'" de = Fa) (b) 

o o T(a +b) 
Also, Tr(a + 1) = ar (a), and T(n) = (n — 1)! if n is a positive integer. 


Euler’s Approximation for Harmonic Sums 


= logn + 0.577... 


Sih 


Ep p Gece 
2 3 


Stirling’s Approximation for Factorials 


n^ 
n! x V2rn (2) 
e 


Miscellaneous Definitions 


Medians and Quantiles Let X have CDF F. Then X has median 
m if F(m) > 0.5 and P(X > m) > 0.5. For X continuous, m satisfies 
F(m) = 1/2. In general, the ath quantile of X is min(z : F(x) > a}; 
the median is the case a = 1/2. 


log Statisticians generally use log to refer to natural log (i.e., base e). 


i.i.d r.v.s Independent, identically-distributed random variables. 


Example Problems 


Contributions from Sebastian Chiu 


Calculating Probability 


A textbook has n typos, which are randomly scattered amongst its n 
pages, independently. You pick a random page. What is the 
probability that it has no typos? Answer: There is a (a — i) 


probability that any specific typo isn't on your page, and thus a 


e 


large, this is approximately e^! = 1/e. 


probability that there are no typos on your page. For n 


Linearity and Indicators (1) 


In a group of n people, what is the expected number of distinct 
birthdays (month and day)? What is the expected number of birthday 
matches? Answer: Let X be the number of distinct birthdays and I; 
be the indicator for the jth day being represented. 


E(I;) = 1 — P(no one born on day j) = 1 — (364/365)” 


E(X) = 365 (1 — (364/365)") |. Now let Y be the 


number of birthday matches and J; be the indicator that the ith pair 
of people have the same birthday. The probability that any two 


E(Y) = (2) /365 . 


By linearity, 


specific people share a birthday is 1/365, so 


Linearity and Indicators (2) 


This problem is commonly known as the hat-matching problem. 
There are n people at a party, each with hat. At the end of the party, 
they each leave with a random hat. What is the expected number of 
people who leave with the right hat? Answer: Each hat has a 1/n 
chance of going to the right person. By linearity, the average number 


n(1/n) —1. 


of hats that go to their owners is 


Linearity and First Success 


This problem is commonly known as the coupon collector problem. 
'There are n coupon types. At each draw, you get a uniformly random 
coupon type. What is the expected number of coupons needed until 
you have a complete set? Answer: Let N be the number of coupons 
needed; we want E(N). Let N = Ni +---+ Nn, where Ni is the 
draws to get our first new coupon, N» is the additional draws needed 
to draw our second new coupon and so on. By the story of the First 
Success, No ~ FS((n — 1)/n) (after collecting first coupon type, there's 
(n — 1)/n chance you'll get something new). Similarly, 

N3 ~ FS((n — 2)/n), and Nj ~ FS((n — j + 1)/n). By linearity, 


1 
E(N) = E(N) +- E(N) = = + — Real 


This is approximately n(log(n) + 0.577) by Euler's approximation. 


Orderings of i.i.d. random variables 


I call 2 UberX's and 3 Lyfts at the same time. If the time it takes for 
the rides to reach me are i.i.d., what is the probability that all the 
Lyfts will arrive first? Answer: Since the arrival times of the five cars 
are i.i.d., all 5! orderings of the arrivals are equally likely. There are 
312! orderings that involve the Lyfts arriving first, so the probability 
312! . 5 
ETE = 1/10 | Alternatively, there are (3) 
ways to choose 3 of the 5 slots for the Lyfts to occupy, where each of 
the choices are equally likely. One of these choices has all 3 of the 


v) 21/0. 


that the Lyfts arrive first is 


Lyfts arriving first, so the probability is 


Expectation of Negative Hypergeometric 


What is the expected number of cards that you draw before you pick 
your first Ace in a shuffled deck (not counting the Ace)? Answer: 
Consider a non-Ace. Denote this to be card j. Let I; be the indicator 
that card j will be drawn before the first Ace. Note that I; = 1 says 
that j is before all 4 of the Aces in the deck. The probability that this 
occurs is 1/5 by symmetry. Let X be the number of cards drawn 
before the first Ace. Then X = Iı + I2 + ... + I48, where each indicator 
corresponds to one of the 48 non-Aces. Thus, 


E(X) = E(I1) + E(12) +... + E(I48) = 48/5 =| 9.6 |. 


Minimum and Maximum of RVs 


What is the CDF of the maximum of n independent Unif(0,1) random 
variables? Answer: Note that for r.v.s X1, X2,..., Xn, 


P(min(Xi,X5,..., X4) > a) = P(X12a, X22 a,..., X, 2 a) 
Similarly, 
P(max(Xi,X23,..., X4) € a) = P(X1€a, X2 € a,..., X. € a) 


We will use this principle to find the CDF of U(,), where 
U(5) = max(U1, U2,...,Un) and Ui ~ Unif(0, 1) are i.i.d. 


P(max(Ui,Us5,...,U,) < a) = P(U, < a,U5 < a,..., Un < a) 
= P(U; < a)P(Us < a)... P(Un < a) 


n 
=|a 


for 0 <a < 1 (and the CDF is 0 for a < 0 and 1 for a > 1). 


Pattern-matching with e” Taylor series 


1 
For X ~ Pois(A), find B(x j Answer: By LOTUS, 


41 
1 oo 1 —Ayk -A œ k+l =À 
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Adam's Law and Eve’s Law 


William really likes speedsolving Rubik’s Cubes. But he’s pretty bad 
at it, so sometimes he fails. On any given day, William will attempt 
N ~ Geom(s) Rubik’s Cubes. Suppose each time, he has probability p 
of solving the cube, independently. Let T be the number of Rubik’s 
Cubes he solves during a day. Find the mean and variance of T. 
Answer: Note that T|N ~ Bin(N, p). So by Adam's Law, 


p(l- s) 


E(T) = E(E(T|N)) = E(Np) = 


Similarly, by Eve's Law, we have that 


Var(T) = E(Var(T|N)) + Var(E(T|N)) = E(Np(1 — p)) + Var(Np) 


»(1—-p)(1—s) pa-s) |p- s)(pM4s(1— p)) 


— T 
s s? 2 


S 


MGF - Finding Moments 


Find E(X?) for X ~ Expo(A) using the MGF of X. Answer: The 
MGF of an Expo(A) is M(t) = 324. To get the third moment, we can 
take the third derivative of the MGF and evaluate at t = 0: 


E(x?) = Ê 


A3 


But a much nicer way to use the MGF here is via pattern recognition: 
note that M(t) looks like it came from a geometric series: 


1 Se EN SA nl t? 
==} (i) = 2059 al 


t 
A n=0 


The coefficient of a here is the nth moment of X, so we have 


E(X”) = 34 for all nonnegative integers n. 
Markov chains (1) 
Suppose X,, is a two-state Markov chain with transition matrix 
0 1 
2-1 (75° 1*5) 


Find the stationary distribution s = (so, s1) of X, by solving SQ = s, 
and show that the chain is reversible with respect to s. Answer: The 
equation 5Q = s says that 

so = so(1— o) + 518 and sı = so(o) + so(1 — B) 


By solving this system of linear equations, we have 


SENS 
| NocrB' acf 


To show that the chain is reversible with respect to s, we must show 
Siqij = Sjqji for all i, j. This is done if we can show s9q01 = 51410: 
And indeed, 


a B 


a 4- B 


s09d01 = = $1910 


Markov chains (2) 


William and Sebastian play a modified game of Settlers of Catan, 
where every turn they randomly move the robber (which starts on the 
center tile) to one of the adjacent hexagons. 


Robbey 


b d 


(a) Is this Markov chain irreducible? Is it aperiodic? Answer: 


Yes to both. 
get from anywhere to anywhere else. The Markov chain is 
aperiodic because the robber can return back to a square in 
2,3,4,5,... moves, and the GCD of those numbers is 1. 


(b) What is the stationary distribution of this Markov chain? 
Answer: Since this is a random walk on an undirected graph, 
the stationary distribution is proportional to the degree 
sequence. The degree for the corner pieces is 3, the degree for 
the edge pieces is 4, and the degree for the center pieces is 6. 
To normalize this degree sequence, we divide by its sum. The 
sum of the degrees is 6(3) + 6(4) + 7(6) = 84. Thus the 
stationary probability of being on a corner is 3/84 = 1/28, on 
an edge is 4/84 = 1/21, and in the center is 6/84 = 1/14. 


(c) What fraction of the time will the robber be in the center tile 
1/14 |. 


The Markov chain is irreducible because it can 


in this game, in the long run? Answer: By the above, 


(d) What is the expected amount of moves it will take for the 
robber to return to the center tile? Answer: Since this chain is 
irreducible and aperiodic, to get the expected time to return we 
can just invert the stationary probability. Thus on average it 


will take | 14 


turns for the robber to return to the center tile. 


Problem-Solving Strategies 


Contributions from Jessy Hwang, Yuan Jiang, Yuqi Hou 


1. Getting started. Start by defining relevant events and 
random variables. (“Let A be the event that I pick the fair 
coin”; “Let X be the number of successes.” ) Clear notion is 
important for clear thinking! Then decide what it is that you’re 
supposed to be finding, in terms of your notation (“I want to 
find P(X = 3|A)”). Think about what type of object your 
answer should be (a number? A random variable? A PMF? A 
PDF?) and what it should be in terms of. 


Try simple and extreme cases. To make an abstract experiment 
more concrete, try drawing a picture or making up numbers 
that could have happened. Pattern recognition: does the 
structure of the problem resemble something we've seen before? 


2. Calculating probability of an event. Use counting 
principles if the naive definition of probability applies. Is the 
probability of the complement easier to find? Look for 
symmetries. Look for something to condition on, then apply 
Bayes’ Rule or the Law of Total Probability. 


3. Finding the distribution of a random variable. First make 
sure you need the full distribution not just the mean (see next 
item). Check the support of the random variable: what values 
can it take on? Use this to rule out distributions that don't fit. 
Is there a story for one of the named distributions that fits the 
problem at hand? Can you write the random variable as a 
function of an r.v. with a known distribution, say Y — g( X)? 


4. Calculating expectation. If it has a named distribution, 
check out the table of distributions. If it's a function of an r.v. 
with a named distribution, try LOTUS. If it's a count of 
something, try breaking it up into indicator r.v.s. If you can 
condition on something natural, consider using Adam's law. 


5. Calculating variance. Consider independence, named 
distributions, and LOTUS. If it's a count of something, break it 
up into a sum of indicator r.v.s. If it’s a sum, use properties of 
covariance. If you can condition on something natural, consider 
using Eve's Law. 


6. Calculating E(X?). Do you already know E(X) or Var(X)? 
Recall that Var(X) = E(X?) — (E(X))?. Otherwise try 
LOTUS. 


7. Calculating covariance. Use the properties of covariance. If 
you're trying to find the covariance between two components of 
a Multinomial distribution, X;, Xj, then the covariance is 
—npip; for i  j. 


8. Symmetry. If X1,..., X, are i.i.d., consider using symmetry. 


9. Calculating probabilities of orderings. Remember that all 
n! ordering of i.i.d. continuous random variables X1,..., X, 
are equally likely. 


10. Determining independence. There are several equivalent 
definitions. Think about simple and extreme cases to see if you 
can find a counterexample. 


11. Do a painful integral. If your integral looks painful, see if 
you can write your integral in terms of a known PDF (like 
Gamma or Beta), and use the fact that PDFs integrate to 1? 


12. Before moving on. Check some simple and extreme cases, 
check whether the answer seems plausible, check for biohazards. 


Biohazards 


Contributions from Jessy Hwang 


1. Don't misuse the naive definition of probability. When 
answering “What is the probability that in a group of 3 people, 
no two have the same birth month?", it is not correct to treat 
the people as indistinguishable balls being placed into 12 boxes, 
since that assumes the list of birth months (January, January, 
January) is just as likely as the list (January, April, June], 
even though the latter is six times more likely. 


2. Don't confuse unconditional, conditional, and joint 
P(B|A)P(A) 
P(B) 
correct to say “P(B) = 1 because we know B happened"; P(B) 
is the prior probability of B. Don't confuse P(A|B) with 

P(A, B). 


probabilities. In applying P(A|B) — , it is not 


3. Don’t assume independence without justification. In the 
matching problem, the probability that card 1 is a match and 
card 2 is a match is not 1/n?. Binomial and Hypergeometric 
are often confused; the trials are independent in the Binomial 
story and dependent in the Hypergeometric story. 


4. Don’t forget to do sanity checks. Probabilities must be 
between 0 and 1. Variances must be > 0. Supports must make 
sense. PMFs must sum to 1. PDFs must integrate to 1. 


5. Don't confuse random variables, numbers, and events. 
Let X be an r.v. Then g(X) is an r.v. for any function g. In 
particular, X?, |X|, F(X), and Ix a are r.v.s. 

P(X? « X|X > 0), E(X), Var(X), and g(E(X)) are numbers. 
X = 2 and F(X) > —1 are events. It does not make sense to 
write f^". F(X)dz, because F(X) is a random variable. It does 


not make sense to write P(X), because X is not an event. 


6. Don’t confuse a random variable with its distribution. Recommended Resources 
To get the PDF of X?, you can’t just square the PDF of X. 
The right way is to use transformations. To get the PDF of 
X +Y, you can’t just add the PDF of X and the PDF of Y. 


The right way is to compute the [convolution] e Introduction to Probability Book 
(http://bit.ly/introprobability) 
7. Don't pull non-linear functions out of expectations. e Stat 110 Online (http://stat110.net) 

E(g(X)) does not equal g(E(X)) in general. The St. e Stat 110 Quora Blog (https://stati10.quora.com/) 

Petersburg paradox is an extreme example. See also Jensen's 

inequality. The right way to find E(g(X)) is with[LOTUS e Quora Probability FAQ (http: //bit.ly/probabilityfaq) 
© R Studio (https://www. xstudio. con) 
e LaTeX File (github. com/wzchen/probability_cheatsheet ) 


Please share this cheatsheet with friends! 


http: //wzchen.com/probability-cheatsheet 


Distributions in R 


Command What it does 
help(distributions) shows documentation on distributions 
dbinom(k,n,p) PMF P(X =k) for X ~ Bin(n, p) 
pbinom(x,n,p) CDF P(X < zc) for X ~ Bin(n, p) 
qbinom(a,n,p) ath quantile for X ~ Bin(n, p) 
rbinom(r,n,p) vector of r i.i.d. Bin(n, p) r.v.s 
dgeom(k,p) PMF P(X =k) for X ~ Geom(p) 
dhyper(k,w,b,n) PMF P(X =k) for X ~ HGeom(w, b, n) 
dnbinom(k,r,p) PMF P(X =k) for X ~ NBin(r, p) 
dpois(k,r) PMF P(X = k) for X ~ Pois(r) 
dbeta(x,a,b) PDF f(x) for X ~ Beta(a, b) 
dchisq(x,n) PDF f(x) for X ~ x2 
dexp(x,b) PDF f(x) for X ~ Expo(b) 
dgamma(x,a,r) PDF f(a) for X ~ Gamma(a, r) 
dlnorm(x,m,s) PDF f(x) for X ~ £N (m, s?) 
dnorm(x,m,s) PDF f(x) for X ~ N (m, s?) 
dt(x,n) PDF f(x) for X ~ t, 
dunif(x,a,b) PDF f(a) for X ~ Unif(a, b) 


The table above gives R commands for working with various named 
distributions. Commands analogous to pbinom, qbinom, and rbinom 
work for the other distributions in the table. For example, pnorm, 
qnorm, and rnorm can be used to get the CDF, quantiles, and random 
generation for the Normal. For the Multinomial, dmultinom can be used 
for calculating the joint PMF and rmultinom can be used for generating 
random vectors. For the Multivariate Normal, after installing and 
loading the mvtnorm package dmvnorm can be used for calculating the 
joint PDF and rmvnorm can be used for generating random vectors. 


Table of Distributions 


Distribution PMF/PDF and Support Expected Value Variance MGF 
Bernoulli P(X=1)=p 
Bern(p) P(X =0)=q=1-p p pq q + pe! 
=F) = k gn—k 
Binomial P(X =k) = (p) pq” 
Bin(n, p) k € {0,1,2,...n} np npq (q + pey” 
Geometric P(X =k) =q*p 
Geom(p) k € {0, 1, 2, ...} q/p q/ p° <a «1 
; " k P(X =n) = (m rm 
Negative Binomial r-1 JP 4 
NBin(r, p) n € {0, 1, 2,...} rq/p rq/ p? up: yr, qe! « 1 
Hypergeometric P(X-k)- ($) [nce A 
HGeom(w, b, n) k € {0,1,2,...,n} L= v Hcr) nB(1— 5 messy 
—Ayk 
Poisson P(X =k)=* ki ; 
Pois(A) kc {0, 1, 2,...} A A eMe -0 
i f(z) = 523 
Uniform b—a " (b—a)? "E 
8 + —a X tef 
Unif(a, b) x € (a,b) = m ea) 
— LÀ o-(2 — 2)2/(207) 
Normal TS evan ee 22,2 
N (nu, 07) x € (—00, 00) n o? eint ^3 
Exponential f(z) = Ae 
Expo(A) x € (0,00) i d r tA 
EE: —Ar1l 
Gamma f(z) = Ta àr) e E- à 
Gamma(a, A) x € (0,00) ` i (xà) QUE 
— I(atb) ,a—1 6-1 
Beta TE) = rero (1 — 2) uos 
ia 
Beta(a, b) x € (0,1) b= ats EpL] messy 
1 = = 2 2 
Log-Normal azeaxt eB He d 
2 
LN (u, 07) x € (0,00) 0 = ento? [2 6?(e7 — 1) doesn’t exist 
1 2-1,—«/2 
Chi-Square waa” e 
xà x € (0,00) n 2n (1— 2t)7"/2, t< 1/2 
T((n-1/2) 2 J) —(n+1)/2 
Student-t vnal'(n/2) rele) 
tn x € (—o00, 00) Oifn>1 


n 


at ifn >2 


doesn’t exist 


